Augmented reality (AR) involves enhancing the perception of a real world environment with supplementary visual and/or audio content, where artificial information is overlaid onto a view of a physical environment. The supplementary content may be projected onto a personalized display device, which may be specifically adapted for AR use, such as a head-mounted display (HMD) or AR-supporting eyeglasses or contact lenses, or alternatively the display screen of a mobile computing device (e.g., a smartphone or tablet computer). The supplementary content is typically presented in real-time and in the context of elements in the current environment.
AR is increasingly utilized in a wide variety of fields, ranging from: medicine (e.g., enhancing X-ray, ultrasound or endoscopic images of the interior of a human body to assist therapeutic and diagnostic procedures); commerce (e.g., allowing a customer to view the inside of a packaged product without opening it); and education (e.g., superimposing relevant educational material to enhance student comprehension); to: military (e.g., providing combat troops with relevant target information and indications of potential dangers); entertainment (e.g., augmenting a broadcast of a sporting event or theatre performance); and tourism (e.g., providing relevant information associated with a particular location or recreating simulations of historical events). The number and variety of potential applications for AR continues to expand considerably.
In order to associate the supplementary content with the real world environment, an image of the environment captured by a camera or image sensor may be utilized, as well as telemetric data obtained from detectors or measurement systems, such as location or orientation determining systems, associated with the camera. A given camera includes various optics having particular imaging characteristics, such as the optical resolution, field of view, and focal length. These characteristics ultimately influence the parameters of the images acquired by that camera, as does the position and viewing angle of the camera with respect to the imaged scene. The measurement systems are inherently capable of supplying a certain level of accuracy or precision, but ultimately have limitations arising from the inherent precision of the various components. Such limitations may also vary depending on the particular environmental conditions (e.g., measurement accuracy may be lower during nighttime, rain or snow, or inclement weather), and may exacerbate over time due to gradual degradation of the optics and other system components. As a result, the position and orientation of the camera as acquired via the associated measurement systems may not correspond exactly to the real or true position and orientation of the camera. Such inaccuracies could be detrimental when attempting to georegister image data for displaying AR content, as it can lead to the AR content being superimposed out of context or at an incorrect relative position with respect to the relevant environmental features or elements.
PCT Patent Application No. WO 2012/004622 to Piraud, entitled “An Augmented Reality Method and a Corresponding System and Software”, is directed to an augmented reality method and system for mobile terminals that involves overlaying location specific virtual information into the real images of the camera of the mobile terminal. The virtual information and also visual information about the environment at the location of the terminal is selected and downloaded from a remote database server, using as well the location (via GPS) and the orientation (via magnetometer, compass and accelerometer) of the mobile terminal. This information is continuously updated by measuring the movement of the mobile terminal and by predicting the real image content. The outline of the captured scene (i.e., crest lines of mountains in the real camera images) is compared with the outline of a terrain model of the scene at the location of the mobile terminal.
U.S. Patent Application No. 2010/0110069 to Yuan, entitled “System for Rendering Virtual See-Through Scenes”, is directed to a system and method for displaying an image on a display. A three-dimensional representation of an image is obtained, and is rendered as a two-dimensional representation on the display. The location and viewing orientation of a viewer with respect to the display (e.g., the viewer's head and/or eye position, gaze location) is determined, using an imaging device associated with the display. The displayed rendering is modified based upon the determined location and viewing orientation.
U.S. Patent Application No. 2010/0110069 to Ben Tzvi, entitled “Projecting Location Based Elements over a Heads Up Display”, is directed to a system and method for projecting location based elements over a heads up display (HUD). A 3D model of the scene within a specified radius of a vehicle is generated based on a digital mapping source of the scene. A position of at least one location aware entity (LAE) contained within the scene is associated with a respective position in the 3D model. The LAE from the 3D model is superimposed onto a transparent screen facing a viewer and associated with the vehicle, the superimposition being in a specified position and in the form of a graphic indicator (e.g., a symbolic representation of the LAE). The specified position is calculated based on: the respective position of the LAE in the 3D model; the screen's geometrical and optical properties; the viewer's viewing angle; the viewer's distance from the screen; and the vehicle's position and angle within the scene, such that the viewer, the graphic indicator, and the LAE are substantially on a common line.
U.S. Patent Application No. 2013/0050258 to Liu et al., entitled “Portals: Registered Objects as Virtualized Personalized Displays”, is directed to a see-through head-mounted display (HMD) for providing an augmented reality image associated with a real-world object, such as a picture frame, wall or billboard. The object is initially identified by a user, for example based on the user gazing at the object for a period of time, making a gesture such as pointing at the object and/or providing a verbal command. The location and visual characteristics of the object are determined by a front-facing camera of the HMD device, and stored in a record. The user selects from among candidate data streams, such as a web page, game feed, video, or stocker ticker. Subsequently, when the user is in the location of the object and look at the object, the HMD device matches the visual characteristics of the record to identify the data stream, and displays corresponding augmented reality images registered to the object.
U.S. Patent Application No. 2013/0088577 to Hakkarainen et al., entitled “Mobile Device, Server Arrangement and Method for Augmented Reality Applications”, discloses a mobile device that includes a communications interface, a digital camera, a display and an augmented reality (AR) entity. The AR entity transmits, via the communications interface, an indication of the mobile device location to an external entity. The AR entity obtains, by data transfer from the external entity via the communications interface, a representation determined on the basis of a number of 3D models of one or more virtual elements deemed as visually observable from the mobile device location, where the representation forms at least an approximation of the 3D models, and where the associated spherical surface is configured to surround the device location. The AR entity produces an AR view for visualization on the display based on the camera view and orientation-wise matching portion, such as 2D images and/or parts thereof, of the representation.
Pritt, Mark D., and LaTourette, Kevin J., “Automated Georegistration of Motion Imagery”, 40th IEEE Applied Imagery Pattern Recognition Workshop (AIPRW), 2011, discloses techniques for georegistering motion imagery captured by aerial photography, based on the registration of actual images to predicted images from a high-resolution digital elevation model (DEM). The predicted image is formed by generating a shaded rendering of the DEM using the Phong reflection model along with shadows cast by the sun, and then projecting the rendering into the image plane of the actual image using the camera model (collinearity equations with radial lens distortion). The initial camera model of the first image is estimated from GPS data and an initial sensor aimpoint. The predicted image is registered to the actual image by detecting pairs of matching features using normalized cross correlation. The resulting camera model forms the initial camera estimate for the next image. An enhanced version of the algorithm uses two predicted images, where a second predicted image consists of the actual image projected into the orthographic frame of the first predicted image.